Low-Rank Gradient Descent

Several recent empirical studies demonstrate that important machine learning tasks such as training deep neural networks, exhibit a low-rank structure, where most of the variation in loss function occurs only few directions input space. In this paper, we leverage structure to reduce high computational cost canonical gradient-based methods gradient descent (GD). Our proposed Low-Rank Gradient Descent (LRGD) algorithm finds an $\epsilon$ -approximate stationary point p-dimensional by first identifying notation="LaTeX">$r ≤ p$ significant directions, and then estimating true at every iteration computing directional derivatives along those r directions. We establish “directional oracle complexities” LRGD for strongly convex non-convex objective functions are notation="LaTeX">$\mathcal{O}(r\ {\rm{log}}(1/\epsilon) + rp)$ notation="LaTeX">$\mathcal{O}(r/\epsilon^2 , respectively. Therefore, when notation="LaTeX">$r\ll provides improvement over known complexities notation="LaTeX">$\mathcal{O}(p\ \rm{log}(1/\epsilon)$ ) notation="LaTeX">$\mathcal{O}(p/\epsilon^2)$ GD settings, Furthermore, formally characterize classes exactly approximately functions. Empirically, using real synthetic data, gains data has absence does not degrade performance compared GD. This suggests could be used practice any setting place